feat: agentx#348
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…ions The head commit message usually describes an unrelated code change; the workflow display name describes the sweep itself.
The failed-run guard required num_requests_total > 0, so a config whose server never came up (total = 0, e.g. dep4 conc32 in run 28617267459) slipped through as a dataless point. Any row explicitly reporting zero successful requests is a failure regardless of how many were issued.
The documented DUMP_DIR mode 500'd on every new surface: the four new tables (agentic_trace_replay, datasets, dataset_conversations, run_datasets) were missing from TABLE_INSERT_ORDER so dumps never carried them, json-provider had no mirrors, and ten routes called getDb() with no JSON_MODE guard. Tables added in FK-safe order; bytea blobs round-trip through dump/load (Buffer JSON encoding, ::bytea decode); agentic_trace_replay lazy-loads like server_logs; mirrors reuse the same pure compute helpers as the SQL paths for version-stale fallbacks; all ten routes gain the standard JSON_MODE branch. Verified end-to-end: dump-mode server serves all ten endpoints 200, byte-identical to Postgres on 9/10 (remaining diffs are pre-existing benchmarks-mirror nuances). Adds 21 mirror tests.
The AgenticTraces default resolved before availability loaded (static SEQUENCE_OPTIONS fallback), so fixed-seq-only models flashed 'Agentic Traces', fired a wasted agentic fetch, then snapped to 1k/1k. New pure resolveEffectiveSequence helper (mirrors default-precisions pattern) returns the real scenario only once availability is known; benchmark fetching gates on the new sequenceResolved flag; non-agentic models fall back to 8k/1k (master's default) when available. Fixes the url-params and historical-trends e2e failures the PR description labels 'pre-existing' — they were caused by this default and now pass with no assertion changes. ttft-x-axis-toggle gets spec-scoped agentic intercepts (shared fixtures have no agentic rows). Verified live: llama70b -> 8K/1K, zero agentic calls, one benchmarks fetch; dsr1 -> Agentic Traces, one fetch.
The public conversation search embedded user input in ILIKE unescaped and uncapped: '%' matched every row and long stacked-wildcard patterns could push Neon to statement timeout (500s). escapeLikePattern escapes backslash-first then %/_ so searches are literal substring matches (now agreeing exactly with the dump-mode mirror's .includes semantics); the route trims and rejects >100 chars with 400 before touching the DB. Live: ?search=%25 30 -> 0 rows; 150-char input -> 400; real searches unchanged. Adds 14 tests.
…overlay-mode e2e Adds the AGENTS.md-required track() calls (agentic_siblings_navigated, datasets_conversations_page_changed, agentic_chart_expanded) to the three untracked interaction clusters, and the mandated overlay-path regression coverage: ttft-x-axis-toggle gains three tests loading an ?unofficialrun= overlay, switching to the ttft x-axis mode (overlay points still render), and asserting the normalized-e2e suppression banner. Cypress 8/8.
Playwright MCP page snapshots contain HTML-entity-escaped class strings; Tailwind 4's auto content detection (which respects gitignore) scanned them and emitted unresolvable mask-image classes, 500ing the dev server. Affects any Playwright-MCP-driven review session incl. the @claude CI review flow.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Agentic ships dark for now: the existing feature-gate mechanism (the same localStorage flag behind the hidden-tabs pattern) now also gates every AgentX surface, default off. One deep chokepoint does most of the work: availableSequences strips agentic-traces when locked, which cascades through the sequence resolver - no agentic default, no scenario entry, no agentic x-axis modes/percentile UI, no agentic tooltips (all downstream of effectiveSequence). The Datasets nav link hides, and the standalone /datasets/* and /inference/agentic/[id] pages notFound() when locked (they have no nav entry and are robots-noindexed). Sitemap/llms.txt already exclude them. API routes and ingest are not gated. Agentic cypress specs seed the gate via unlockAgenticGate(). Verified both states live: locked = 8k/1k default with zero agentic DOM + 404s; unlocked = everything restored.
agentic-point-time-series: the point-count assertions predated the time-boundary phase slicing — sliceTimelineByPhase puts every request with start >= the profiling boundary in the profiling window, so the warmup-labeled r5 legitimately lands there and cancelled/null-metric filtering yields 6 interactivity/TTFT points and 8 E2E points (traced to phase-slice.ts + time-series-math.ts, not just observed). No product regression. gpu-compare-agentic-detail: shared fixtures carry no agentic rows, so the flow could never render; spec-scoped intercepts (availability, benchmarks, trace-availability) now exercise the tooltip -> View charts link without touching shared fixtures. Full e2e suite: 449/449. These specs were failing all four CI shards on every push of this branch.
…guards, conv-id encoding, stale intvty, phase filter) - Chart-layer offload dedup: useChartData's latest-date-per-group key and mergeRunScopedRows' claim key now include offload_mode (?? 'off'), completing the SQL-layer fix - a later-dated offload sweep no longer drops the other variant's series or claims its base rows. - PARAM_DEFAULTS.i_seq '' so an explicit 8K/1K pick survives share URLs instead of stripping and reloading as the agentic default; moved PRE_AVAILABILITY_SEQUENCE below imports (code-quality flag). - Shared isPersistedBenchmarkId guard (integer > 0) at every agentic link/id-collection site: no /agentic/NaN or /agentic/0 links, overlay-only views skip the derived-metrics fetch instead of 400ing, and agentic/[id] notFound()s on invalid ids. - Conversation ids: encode once at link producers, decode exactly once (removed double decodeURIComponent in page + route); ids with % / # ? now round-trip. - ETL: missing/zero/invalid ITL now deletes the artifact-provided *_intvty key instead of passing p(1/ITL) values through; same fix on the documented overlay-path mirror in benchmark-transform. - Gantt timeline now phase-slices by time boundary like the per-point charts (sliceTimelineByPhase), so both tabs agree on request sets; spec expectation updated with producing-logic justification. - gitignore **/.next-* (secondary dist dirs from multi-server testing). app 2381 + db 399 unit tests, full e2e 449/449.
The mode row (TTFT / E2E Latency / Normalized E2E / Interactivity / Session Time / Prefill TPS) was hand-rolled chunky pills, visually disconnected from every other control. It now reuses SegmentedToggle - the repo's actual tab idiom (10+ call sites incl. the adjacent View Mode toggle) - sized up via buttonClassName since this is a primary control. Testids, track() events, tablist/tab aria semantics, and agentic-only visibility unchanged; net -12 lines. Cypress ttft-x-axis-toggle 8/8.
Replaces the SegmentedToggle pills with an accent-underline tab strip
(revived the unused Radix-backed ui/tabs.tsx primitive). Active tab
gets the repo's established nav-tab underline token (border-secondary
in light, dark:border-primary), plus a bg-muted/60 active fill so the
minecraft theme — whose global 'button { border: 2px !important }'
override suppresses the underline — still distinguishes the active tab.
Testids, track() events, agentic-only visibility, and Radix a11y
(role=tab/aria-selected/keyboard) all preserved; SegmentedToggle stays
for the adjacent view-mode toggle. cypress 8/8, vitest 2381/2381.
Drop the card-fill + white active text; adopt tab-nav.tsx's flat underline-strip recipe verbatim — active = accent text (text-secondary dark:text-primary) + matching border-b-2 underline, no background, inactive = muted-foreground with border-only hover. The x-axis mode row now reads identically to the dashboard's top section tabs. cypress 8/8, vitest 2381.
Reverts two of this PR's default-flips per product decision: high contrast (i_hc) and parallelism/advanced labels (i_advlabel) now default off. InferenceContext drops defaultHighContrast:true and flips the advlabel init to === '1'; both write-backs now encode ON as '1' / OFF as '' (matching i_linelabel), consistent with the unchanged PARAM_DEFAULTS so bare links render both off and i_hc=1/i_advlabel=1 still enable them. Specs updated; 62/62 affected e2e green.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c9c8074. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds agentic trace-replay benchmarks (AgentX) to the dashboard: closed-loop replays of real multi-turn agent conversations (with subagent fan-out), alongside the existing fixed-seq-len sweeps. History is squashed into the logical commit groups below.
Database schema (
feat(db): agentic benchmark schema)New migrations (
007_agentic.sql…009_dataset_request_stats.sql):benchmark_resultsextended for agentic rowsbenchmark_type = 'agentic_traces';isl/oslnullable (traces have no fixed sequence length)offload_modecolumn added to the unique key (nulls not distinct) — the same config can run with KV offload on/offtrace_replay_idFK into the new sidecar tablemetricsJSONB gains trace-derived keys: full latency percentile ladders (mean/median/p75/p90/p95/p99/p99.9×ttft/tpot/itl/e2el),*_intvty(interactivity = 1/ITL), per-GPU throughputs,gpu_kv_cache_usage_pct, server/theoretical cache-hit rates, token totalsagentic_trace_replay— one row per benchmark point, keeps 100 MB+ payloads out of the hot query pathprofile_export_jsonl_gz(every request the load generator sent) andserver_metrics_json_gz(vLLM/SGLang Prometheus scrape summary)chart_series,request_timeline,aggregate_statsdatasets/dataset_conversations— the source trace datasets (HF id, slug, variant) and per-conversation structure: turn/subagent tree, token counts, timing, cache-hit distributionsrun_datasets— links each workflow run to the dataset it replayed, so timeline → dataset deep links resolvelatest_benchmarksmaterialized view updated to include agentic rows (one run per line)Time-series & per-request data ingested per run
chart_series(1 Hz server-side time series, one point per second of the run): KV-cache utilization (aggregate, per-DP-rank/engine, and host/CPU-offload for hicache), prefix-cache hit rate and hit-rate tokens/sec, queue depth (running/waiting), prefill and decode tokens/sec, prompt tokens by source — plus the same series grouped by server source for disagg runsrequest_timeline(one record per HTTP request, ns-resolution): conversation id + turn index, worker id, subagent depth, warmup/profiling phase, credit-issued/start/ack/end timestamps, TTFT/TPOT, ISL/OSL, cancellation flag, and raw-source provenance (srcTrace/srcOuter/srcInner/srcKind) mapping every replayed request back to the exact original dataset requestaggregate_stats— percentile envelopes precomputed across sibling configs so the cross-config aggregate views don’t re-stream blobsCHART_SERIES_VERSION,REQUEST_TIMELINE_VERSION,STATS_VERSION): version-mismatched rows recompute from the raw blob on demand, anddb:backfill-*CLIs re-materialize in bulk after an algorithm changeRemaining commit groups
chore:deps/toolchain — stream-json (streaming 100 MB+ blobs withoutERR_STRING_TOO_LONG), adm-zip, audit overridesfeat(db):ETL — maps aiperf artifacts across all three agg-schema generations (v3: nested request/server metrics, pre-inverted interactivity,cluster:-scoped hw ids,results/server.log); computes the three JSONB payloads at ingestfeat(db):query layer + CLIs — fast-path queries over precomputed JSONBs with version-checked blob fallback (guarding Neon’s 64 MiB response cap), shared backfill runner behind the fourdb:backfill-*scripts,ingest-weka-datasetfor loading source datasetsci:ingest workflow —repository_dispatch: ingest-agentic-resultsso the main InferenceX repo triggers ingestion after an AgentX sweep (mirrors the fixed-seq-len flow; 60-min timeout for blob uploads; Slack alert when a run references a dataset missing fromdatasets)feat(api):endpoints + hooks — bulk-id v1 routes (aggregates, derived metrics, request timelines, server metrics, histograms, siblings, datasets) with shared route/hook factories for a uniform error contract and caching; matching React Query hooksfeat(datasets):dataset browser — dataset list/detail with token & cache-hit distributions; per-conversation flamegraph showing the turn/subagent timing tree; accepts?turn/raw/inner/sadeep-link params to highlight the exact requestfeat(agentic):per-point detail — summary cards; Gantt request timeline (per-conversation or per-worker rows, subagent/aux lane nesting, stable row colors across phase toggle, shift+scroll zoom, click-through to the original dataset request); rolling/cumulative time-series charts (interactivity, TTFT/E2E, KV usage, throughput, in-flight ISL/OSL); cross-config aggregates. Exists to debug why identical configs diverge under sustained loadfeat(inference):dashboard integration — Agentic Traces as a scenario in the main chart: trace-derived x-axis modes, agentic-aware rooflines/tooltips/labels, click-through to point detail; works for official runs and?unofficialrun=overlaystest:coverage + docs — Cypress e2e/component specs for timeline, time-series, datasets, and dashboard integration; data-pipeline docsDiff breakdown
~24% of the diff is test coverage.
Verification
url-params.cy.ts(2): Historical Trends tab defaults to Agentic Traces, which has no trend data, so the legend/high-contrast toggle never rendersagentic-point-time-series.cy.ts(3): stale point-count/label expectations from earlier chart changesNote
High Risk
Large surface area (~21k LOC) touching DB ingest, new APIs with blob caching, and core global filter/scenario resolution; incorrect gating or cache versioning could mislabel charts or serve stale agentic payloads in production.
Overview
This PR ships AgentX end-to-end: agentic trace-replay benchmarks on the main inference chart, deep per-point views, a dataset browser, v1 APIs over precomputed trace blobs, and CI/operator docs for ingesting blob-heavy runs.
Inference dashboard renames the sequence control to scenario (
scenario-selector,i_sequnchanged) and adds Agentic Traces as a first-class scenario with new x-axis modes (interactivity, TTFT, E2E, normalized E2E), percentile selection, legend points table drill-down to/inference/agentic/[id], and GPU tooltip View charts links. Global filters gainsequenceResolvedso benchmarks and labels wait for real availability (no agentic flash/fetch for fixed-seq-only models). Agentic UI is behind a localStorage feature gate (AgenticGate, CypressunlockAgenticGate); line labels default off; URL tests coveri_hc=0and default toggle states.New product surfaces:
/datasetsregistry with distributions and searchable conversations; per-conversation flamegraphs (timing, parallel subagent brackets) with deep links from the request timeline;/inference/agentic/[id]detail (gated, positive id only). API layer adds sharedid-routesfactories, dataset CRUD/list routes (search length cap, single-decodeconvIdcontract), and version-derived blob cache keys for aggregates, timelines, histograms, and server metrics.Pipeline & ops:
ingest-agentic-resultsworkflow (60m timeout, multi DB target, Slack on unmapped entities/missing datasets);.claude/agents/ingest.mdrunbook;docs/data-pipeline.mdnotes orchestrator metric adapters andrun_datasetsprovenance frommetadata.dataset. Unofficial-run overlays now carrybenchmark_type/offload_modeand syntheticid: 0so trace-backed features stay disabled for overlays as intended.Reviewed by Cursor Bugbot for commit c9c8074. Bugbot is set up for automated code reviews on this repo. Configure here.